Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add aarch64 manylinux wheels #2425

Merged
merged 6 commits into from
Jun 18, 2024
Merged

feat: add aarch64 manylinux wheels #2425

merged 6 commits into from
Jun 18, 2024

Conversation

mayeut
Copy link
Contributor

@mayeut mayeut commented Jun 15, 2024 •
edited
Loading

Summary

Description

This PR continues work from #2103 (I've never removed that branch and rebased on master once in a while so that's the same branch in this PR).

cherry-picked commits from #2126 for missing test dependency in manylinux.

main differences between this PR and #2224:

  • This PR does not build ppc64le & s390x.

  • This PR does not build musllinux wheels.

  • This PR adds tests for architectures running QEMU:
    While the tests needed some modifications in order to pass, IMHO, it's best to check than what can be tested does work as intended.
    Running with QEMU however hides some bugs (thinking about /proc/cpuinfo and cpu related stuff here) that actually do exist or comes with its own bugs (s390x and /proc endianness issue is most likely a qemu issue).
    aarch64 will most likely be able to run natively by the end of the year.

  • This PR does not add Windows ARM64 wheel
    It could be added without any tests but I'd rather see Microsoft add Windows ARM64 runners to GitHub if they want to get some traction for their new platform (hopefully by the end of the year).

  • This PR does not add macOS universal2 wheel
    My personal opinion is that it's not useful as long as you provide both x86_64 & arm64 wheels (which is the case).
    They do take more space for the end user, pip will always choose the most specific one so publishing 3 will change nothing for almost all users. Remain people packaging universal2 python apps, and, IMHO, this should be handled by the tools creating such apps.

  • This PR feels simpler regarding the change of workflow

Regarding tests that are failing on real arm64 hardware but hidden by QEMU testing:

  • Using docker on macOS M1:
======================================================================
FAIL: psutil.tests.test_system.TestCpuAPIs.test_cpu_freq
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/psutil/tests/test_system.py", line 609, in test_cpu_freq
    assert ls, ls
AssertionError: []

======================================================================
FAIL: psutil.tests.test_linux.TestSystemCPUCountCores.test_method_2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/psutil/tests/test_linux.py", line 820, in test_method_2
    self.assertEqual(meth_1, meth_2)
AssertionError: 10 != None

======================================================================
FAIL: psutil.tests.test_linux.TestSystemCPUFrequency.test_emulate_use_second_file
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/psutil/tests/test_linux.py", line 845, in test_emulate_use_second_file
    assert psutil.cpu_freq()
AssertionError

======================================================================
FAIL: psutil.tests.test_linux.TestSystemNetIfAddrs.test_ips
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/psutil/tests/test_linux.py", line 1020, in test_ips
    self.assertEqual(addr.address, get_mac_address(name))
AssertionError: '00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00' != '00:00:00:00:00:00'
- 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
+ 00:00:00:00:00:00

======================================================================
FAIL: psutil.tests.test_system.TestCpuAPIs.test_cpu_freq
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/psutil/tests/test_system.py", line 609, in test_cpu_freq
    assert ls, ls
AssertionError: []
======================================================================
FAIL: psutil.tests.test_system.TestSensorsAPIs.test_sensors_temperatures
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/psutil/tests/test_system.py", line 942, in test_sensors_temperatures
    self.assertGreaterEqual(entry.high, 0)
AssertionError: -273.15 not greater than or equal to 0
======================================================================
FAIL: psutil.tests.test_linux.TestSystemCPUCountCores.test_method_2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/psutil/tests/test_linux.py", line 820, in test_method_2
    self.assertEqual(meth_1, meth_2)
AssertionError: 4 != None
======================================================================
FAIL: psutil.tests.test_linux.TestSystemCPUFrequency.test_emulate_use_second_file
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/psutil/tests/test_linux.py", line 845, in test_emulate_use_second_file
    assert psutil.cpu_freq()
AssertionError

@giampaolo
Copy link
Owner

Thank you Matthieu.

I added other platforms as well (ppc64le & s390x) [...] and musllinux

Please, let's just add aarch64 wheels for now (the ones with the highest demand). There's too many skipIf(...) which seems are due to emulation. Just so you know: my main concern here is keeping things as simple and maintainable as possible. Above all, we shouldn't introduce new random test failures (we already have a lot), and producing wheels / releases should be a smooth process.

mayeut and others added 3 commits June 16, 2024 08:22
@mayeut
feat: add linux aarch64 wheel
f16bfe9 
Signed-off-by: mayeut <[email protected]>
@ben9923 @mayeut
Run ifconfig tests in Python 3 CI
d7f4c4f 
Python 3 manylinux CI image does not have net-tools (hence ifconfig)
pre-installed, so tests using it were skippied altogether.

Signed-off-by: Ben Raz <[email protected]>
@mayeut
fix(tests): aarch64 does not report mhz in /proc/cpuinfo
759971b 
Signed-off-by: mayeut <[email protected]>
@mayeut mayeut force-pushed the abi3-linux-aarch64 branch from aaa0606 to 6aa80d3 Compare June 16, 2024 06:51
@mayeut mayeut changed the title feat: add aarch64/ppc64le/s390x manylinux wheels and musllinux wheels feat: add aarch64 manylinux wheels and musllinux wheels Jun 16, 2024
@mayeut
Copy link
Contributor Author

mayeut commented Jun 16, 2024 •
edited
Loading

Please, let's just add aarch64 wheels for now (the ones with the highest demand)

I removed ppc64le & s390x (there's one demand from IBM #2210 for ppc64le).

I kept musllinux which are I think the second highest demand with 1 related issue & 3 opened PR.
If you don't want to keep musllinux, just drop the commit 445b35d.
Please note that 6f55304 is related to the addition of musllinux which comes with an up-to-date version of free/vmstat. Even if not building musllinux, this one will eventually come back as an issue once glibc distros get updated.

There's too many skipIf(...) which seems are due to emulation. Just so you know: my main concern here is keeping things as simple and maintainable as possible.

There are now less with s390x being dropped. I however understand your concern. Given GHA will most likely provide a native aarch64 before the end of the year, the QEMU test commit might be dropped completely 6aa80d3.

Above all, we shouldn't introduce new random test failures (we already have a lot), and producing wheels / releases should be a smooth process.

Agreed, that's why I disabled test_memleaks on those architectures. It was the only flaky one as far as I saw (and memleaks should mostly be independent of the processor architecture and tested otherwise).

Please feel free to force-push, drop commits, ... as you see fit. I have backups / alternatives to this work in my fork.

@giampaolo
Copy link
Owner

If you don't want to keep musllinux, just drop the commit 445b35d.

It seems I cannot modify your PR (I can push but can't see the changes reflected in here).
Can you please remove musl wheels and update the README? The rest looks good.
Also one minor thing. Is there a reason why it's called QEMU_USER instead of QEMU?

@mayeut
Copy link
Contributor Author

mayeut commented Jun 18, 2024

It seems I cannot modify your PR (I can push but can't see the changes reflected in here).

weird...
anyway

Also one minor thing. Is there a reason why it's called QEMU_USER instead of QEMU?

QEMU provides full-system emulation or user-mode emulation. Here, we only use user-mode emulation.

@mayeut mayeut changed the title feat: add aarch64 manylinux wheels and musllinux wheels feat: add aarch64 manylinux wheels wheels Jun 18, 2024
@mayeut mayeut changed the title feat: add aarch64 manylinux wheels wheels feat: add aarch64 manylinux wheels Jun 18, 2024
@mayeut mayeut force-pushed the abi3-linux-aarch64 branch from 6aa80d3 to 06585ec Compare June 18, 2024 18:51
@mayeut
Copy link
Contributor Author

mayeut commented Jun 18, 2024

update the README?

I guess you meant HISTORY ?

@mayeut mayeut force-pushed the abi3-linux-aarch64 branch from 06585ec to 91cf7db Compare June 18, 2024 18:53
@giampaolo
Copy link
Owner

Yes, sorry. 😄

@giampaolo giampaolo merged commit 5b30ef4 into giampaolo:master Jun 18, 2024
18 checks passed
@hoechenberger hoechenberger mentioned this pull request Jun 18, 2024
Open
7 tasks
@mayeut mayeut deleted the abi3-linux-aarch64 branch June 18, 2024 20:58
@shawaj shawaj mentioned this pull request Jun 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

ARM64 support [Linux][aarch64] Wheel support for aarch64 Linux [Linux] Publish armv7l & aarch64 wheels
3 participants
@mayeut @giampaolo @ben9923